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ABSTRACT 


In  this  note  we  examine  the  distribution  function  of  the  random 
00 

variable  Z  =  £  X.  D  1  when  the  random  variables  X  ,X  ,  ...  form  a 

1-1 1  12 

stationary  ergodic  sequence  and  each  X  takes  on  integer  values  running 
from  0  to  D-l.  We  show  that  the  distribution  function  will  be  either  a 
step  function,  or  a  continuous  and  purely  singular  function,  or  else  a 
linear  function  of  its  argument.  The  form  of  the  distribution  function 
is  related  to  the  entropy  rate  of  the  random  sequence.  We  discuss  this 
relationship  and  also  the  relationship  of  entropy  rate  to  the  concept  of 
Hausdorf f  dimension . 
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SECTION  I 


I 


INTRODUCTION 

Let  ^>^2’’"'  a  stationary  ergodic  sequence  of  random  variables 
defined  on  a  probability  space  (Q,A,P)  and  mapping  into  the  set 
{0,1, • • • ,D-l}.  In  particular,  we  may  envision  fl  as  the  set  of  all 
sequences  CO  =  ,  *  *  *  )  with  X^CtO)  =  and  CO^  e  {0,1 , • • • ,D-l} . 

To  avoid  trivialities  we  assume  that  D  ^  2  and  P(w:  X^(co)=i)  >  0  for 
i  =  0,1,- • • ,D-1 . 

00 

The  random  variable  Z  =  Z  X,  D  1  may  be  used  to  transfer  the 

i*l  L 

probability  structure  of  the  random  sequence  to  the  unit  interval;  each 
point  coeQ  is  mapped  by  Z  into  a  point  Z(to)  satisfying  0  £  Z(c o)  £  1. 

In  this  note  we  study  the  distribution  function  of  Z,  F(a)  = 

P(00:  Z(o))  <  a).  We  show  (Theorem  1)  that  either:  F  is  a  step  function 
with  k  jumps  of  height  1/k,  or  F  is  continuous  and  purely  singular,  or 
else  F(a)  =  a;  F(a)  =  a  if  and  only  if  the  random  variables  are 
independent  and  uniformly  distributed.  Harris  [5]  has  proved  an 
analogous  theorem  using  different  techniques.  Our  method  of  proof 
utilizes  Information  Theory  and  illustrates  some  interesting  relation¬ 
ships  between  the  entropy  rate,  H,  of  the  random  sequence  and  the  form 
of  the  distribution  function.  In  particular,  H  is  maximal;  that  is 
H  *  log  D  =  1  (all  logarithms  will  be  taken  to  the  base  D),  if  and 
only  if  F(a)  **  a.  If  0  <  H  <  1  then  F  is  purely  singular  and  continuous. 
If  F  is  a  step  function  then  H  a  0.  Presumably  there  exist  ergodic 
sequences  for  which  H  =  0  and  F  is  continuous  and  purely  singular 
though  we  have  not  been  able  to  construct  any. 
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In  the  proof  of  Theorem  1  we  define  a  set  E  which  is  in  some  sense 
the  support  of  the  distribution  function  F  when  it  is  regarded  as  a  * 

measure.  Thus,  if  F  is  a  step  function,  E  consists  of  the  points  of 
discontinuity  of  F;  if  F(a)  =  a  then  E  *  [0,1];  if  F  is  continuous  and 
purely  singular  /i(E)  =  0  but  ji(F(E))  =  1  (fi  is  Lebesque  measure).  We 
show  in  Section  IV  that  the  Hausdorff  dimension  of  the  set  E  is  equal  to 
H.  This  extends  a  theorem  proved  by  Kinney  [7]  for  ergodic  Markov 
chains . 
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p 


* 


* 


» 


>•%» 


* 


SOME  INFORMATION  THEORETIC  PRELIMINARIES 

We  review  briefly  some  pertinent  facts  from  Information  Theory. 

It  will  be  convenient  to  introduce  the  symbol  P(a^,***,an)  as  a 

shorthand  notation  for  P(oo:  X.  (to)  =  a,,-’-,X  (w)  *  a  )  where  each 

1  1  n  n 

a^e  {0,1, • • • ,D-l}  for  i  =  l,-**,n.  Also,  as  noted  earlier,  all 
logarithms  will  be  taken  to  the  base  D. 

To  the  process  [PjX^rn  £  l]  we  associate  a  set  of  non-negative 
numbers . 

H  =  -  E*  P(a1,***,a  )  log  P(a..,**',a  )  . 

n  lnln 

al*  * ’ * ,an 

We  mean  E  to  signify  that  the  summation  is  carried  out  only 

a-  , • • •  ,a 
1  n 

over  those  n- tuples  which  have  a  positive  probability  of  occurrence; 
the  number  of  summands  is  clearly  £  Dn.  The  numbers  which  are 
termed  (n-fold)  entropies  satisfy  the  following  simple  inequalities: 

(a)  £  log  D  -  1 

=  1  if  and  only  if  P(U):  X^(co)  =  j)  =  1/D  for  j  ■  0,***,D-1 

(b)  H-  ^  H  -  H  -  ^  H  -  -  H  n  =  2 ,3 ,  •  •  • 

N  1  n  n-1  n+1  n 

H,  =  H  -  H  ,  for  all  n  ^  2  if  and  only  if  the  random 
1  n  n-1 

variables  are  independent. 

<‘>  Vi  -  H„  £  0 

Hn+1  ‘  Hn  “  0  and  on]Ly  if  P(ai>  *‘’>an+l^  ^  0  imPlies 

that  P(a^ ,  •  •  •  ,an+1)  ■  P(a1,'",an) 
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These  inequalities  may  be  derived  by  judicious  application  of  the 

X“  1 

following  inequality  which  is  valid  for  x  ^  0  :  log  x  £  ; 

x  —  1 

log  x  =  if  and  only  if  x  *  1  where  In  =  log^.  For  purposes  of 
illustration  we  sketch  the  proof  of  (b). 


P(ai,---,an)P<V---,anfl> 


H  +H  - 2  H  ■  Z  PCa-.-'-.a^.)  log  . 

"+1  n’1  "  a,,---,a  2  nfl  p<V"><Wl)P(a2>"'>an) 

1  n+1 


1  * 


ai»”‘»Vi 


p(al » • • *  >an+i} 


P(ai,---,an)P(a2,...,an+1) 


P(al * '  ’ ,an+l)P(a2’' ’ * ,an^ 


1 

InD 


*  P^a? ’ *  *  * ,3n+l ^  * 

S  P(a„--,0  S  p(a1.-",an>-1 

a2  ’ ' *  ,an+l  2  n  al 


1 

InD 


a2 ’ *  ’ an+l 


P(a2, 


,anf 1 ^  -  1 


In  order  for  equality  to  hold  throughout  it  is  necessary  and  sufficient 
that  p(a^> ‘  “  >an+1)  t  0  imply 

p(al  >  •  *  *  >an+i^  p(a2  >  ’  *  ’  >an)  =  p(a1,,“,an)  P(a2,--- ,an+1)  (n  2:  1) 

This  is,  however,  a  necessary  and  sufficient  condition  for  the  random 
variables  to  be  independent. 

From  (a),  (b)  and  (c)  we  deduce  that  the  sequence  {(H  -  ^)}  is 

monotone  non- increasing,  and  bounded  between  0  and  1.  Thus  the  sequence 

has  a  limit  which  we  designate  by  H: 

H*  lim  (H  -  H  ) 
n  n-  i 
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* 


Since 

n 

lim  (H  -  H  .)  «  lim  1  £  (H.-  H.  ) 

n  n-l  “  .  o  i  - 1 

n-^oo  n-*®njB2JJ 

it  follows  readily  that 

H  =  lim  Hn 

n  -»  00  n 

Accordingly  we  term  H  the  entropy  rate  of  the  process. 

Remarks:  1)  The  results  of  this  section  are  valid  for  sequences 

which  are  stationary  but  not  ergodic . 

2)  Observe  that  0  £  H  £  1  ;  H  =  1  if  and  only  if 
the  random  variables  are  both  independent  and 
uniformly  distributed. 
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SECTION  III 


THE  FORM  OF  THE  DISTRIBUTION  FUNCTION 

It  will  be  convenient  to  define  a  shift  operator  T  on  the 
space  Q  by  the  rule  (Tco)^  =  Uu  .  For  a  unilateral  stationary 
sequence  of  random  variables  T  is  a  measurable,  measure  preserving, 
non- invertible  transformation. 

Lemma  1 

Suppose  there  exists  a  point  ueQ  such  that  P(u)  «  e  >  0.  Then 
e  *  1/k  for  some  positive  integer  k,  F  is  a  step  function  with  k  jumps 
of  height  1/k  and  H  =  0. 

Proof 

Designate  the  characteristic  function  of  the  set  {u}  ,  by  . 
Then  by  the  Birkhoff  ergodic  theorem 

n-1 

lim  l  EM  (TJu)  =  P(u)  =  e  >  0  . 

n  -►  <»  n  j  =0 

But  M  (T^u)  =  0  unless  T^u  =  u.  Thus  there  must  exist  some  smallest 
u 

positive  j,  say  j  =  k,  such  that  T^u  =  u.  It  follows  readily  that 

e  *  1/k,  that  the  distinct  points  in  Q  with  non-zero  probability  are 

k-1  k-1 

u,  Tu,  •  *  •  ,  T  u  ,  and  that  P(u)  =  P(Tu)  =  •••  =  P(T  u  )  »  1/k. 

Hn 

For  n  ^  k  H  =  log  k  therefore  H  =  lim  —  =0. 

n  ^  n 

n  ->  00 

Lemma  2 

If  P(co)  =  0  for  every  O)  then  F  is  continuous. 

Proof 

A  probability  measure  is  countably  additive.  Consequently 
P(C0  :  Z(co)  ^  a)  is  a  right  continuous  function  of  a  and 


4 


# 


% 
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P(c 0  :  Z(o i>)  <  a)  is  a  left  continuous  function  of  a.  Thus 
P((0  :  ZCw)  ^  a)  -  P(tc  :  Z(w)  <  a)  ■  P(40  :  Z(oo)  ■  a)  =  0  since 
there  are  at  most  two  points  in  Cl  which  can  be  mapped  by  Z  into  a 
and  each  of  these  has  0  probability. 

Remark  3)  F  will  be  strictly  monotone  increasing  if  and  only  if 
P(a^,**',an)  >0  for  every  finite  sequence  a^,'**,a 
of  elements  chosen  from  {0,1, • • • ,D-l} . 

Lemma  3 

The  random  variables  are  independent  and  uniformly  distributed 
if  and  only  if  H  =  1.  In  this  case  F(a)  =  a. 

Proof 

If  the  random  variables  are  independent  and  uniformly 
distributed  then  F(g)  =  g  at  each  D-adic  rational  point,  g.  The 
D-adic  rational  points  are  dense  in  the  unit  interval  and,  by 
Lemma  2 ,  F  is  continuous.  Hence  F(a)  =  a  for  all  0  £  a  £  1.  It 
was  observed  in  Remark  2  that  H  =  1  if  and  only  if  the  X  are 
independent  and  uniformly  distributed. 

Theorem  1 

The  distribution  function  F  is  one  of  the  following  3  types: 

(1)  a  step  function  with  k  jumps  of  height  1/k  where  k  is 
a  positive  integer, 

(2)  continuous  and  purely  singular, 

(3)  F(a)  **  a.  0  £  a  £  1. 

F  is  of  type  (3)  if  and  only  if  H  =  1.  If  F  is  type  (1)  then  H  ■*=  0. 

\ 

If  H  >  0  F  is  continuous.  (Thus  if  0  <  H  <  1  then  F  is  type  (2).) 
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Proof 


4 


It  follows  from  Lemmas  1  and  2  that  either  F  is  type  (1),  in 
which  case  H  =  0,  or  F  is  continuous.  If  F  is  continuous  and 
H  =  1  then,  by  Lemma  3,  F(a)  =  a.  It  remains  to  show  that  if  F 
is  continuous  and  0  £  H  <  1,  then  F  is  purely  singular.  It 
suffices  to  exhibit  a  set  E  for  which  fi(E)  =  0  and  u(F(E))  «  1. 

Given  any  x  e  [0,l]  and  any  i  ^  1  we  choose  the  unique  pair 
of  non-negative  numbers  =  £^(x)  anc*  ^  =  r|^(x)  such  that 
(\(x)  +  T)^  (x )  =  D  *  and  D^ac-T^OO)  and  D^Cx+^Cx))  are  both  non¬ 
negative  integers . 

Consider  the  set 

log[F(x+(\)  -  F(x-r|i)] 


E  =  4  x  :  lim 

i  -4  CD 


log[q+  T1-] 


=  H 


and  note  that  for  each  xeE  and  each  e  >  0  there  exists  a  number 
N(x,e)  such  that  if  i  >  N(x,e)  then 


D-i(H+e)  ^  pCx+q)  -  F(x-r)i)  £  D~ 1 (H_e ) 


If  H  <  1  we  can  choose  e  >  0  so  that  H  +  e  <  1. 
Clearly  then  for  each  xeE  and  i  >  N 


F (*-<-;.)  -  F(x-nt)  __  ol(1.„.c) 

Ci  + 


Since  F  is  a  bounded  monotone  increasing  function  it  must  have 
a  finite  derivative  almost  everywhere  in  the  sense  of  Lebesque. 

(See  McShane  [9],  Thm  34.2,  p.  202.)  Thus  (jl( E)  =  0. 


# 


«» 
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For  ucQ  set  [u]  ■  {co efl  :  co,  =  u.  •  •  •, co  =  u  } 

n  1  1’  ’  n  nJ 

-log  P([to]  ) 

and  let  fi  ■  {co  :  lim  - —  -  H]  . 

n  -»  oo 

By  the  strong  form  of  McMillan's  theorem  (Breiman  [3]  and  [4]) 
P(Q  )  =  1.  The  identity 

F(Z(u>)  +  Ct>  '  F (Z (co)  -  ni)  =  P([to]jL) 

implies  that  if  ooeO  then  Z(td)eE.  Thus 

/i(F(E))  =  P(to  :  Z(co)eE)  £  P(n*)  =  1  . 

McMillan's  theorem  also  implies  that  if  H  >  0  then  P(co)  =  0  for 
every  CO  eft.  In  this  case  then,  by  Lemma  2,  F  must  be  continuous. 


* 


% 
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SECTION  IV 


ON  THE  HAUSDORFF  DIMENSION  OF  E 


In  this  section  we  show  that  the  Hausdorff  dimension  of  the  set  E 
constructed  in  Theorem  1  is  equal  to  H.  This  is  in  fact  an  immediate 
consequence  of  a  general  theorem  due  to  Billingsley  [2.  Thm  2.4].  'We 
shall  give  a  direct  proof,  however,  establishing  in  the  process  Holder 
conditions  on  F.  The  techniques  used  are  similar  to  those  of  Kinney  [7] 
and  Kinney  and  Pitcher  [8]. 

Lemma  4 

If  xeE  then  for  any  a.  >  0 

lim  [F(x+h)  -  F(x-h)]  (2h)  =  co 

h  -  0 


Proof 


-(H-WO 


The  proof  follows  by  observing  that  we  can  choose  a  positive 
6  <  Oi  and  i  so  large  that 

[F(x  +  Ci  +  Tii)  -  F(x  -  Q±  -  r|i)3  [2(q+  rii)3 

,u  .  -  (H4a) 

*  2'(IH°°  [F (x  +  £.)  -  F(x  -  n.)]  [C-+  rii3 


^  2-(H-to)  Di(Q!-6)  ^  ^  _ 

Lemma  5 

For  every  OL  >  0  there  exists  a  set  ACE  such  that  ji(F(A))  =  1 

and  for  all  xeA  _H4rY 

lim'  [F(x+h)  -  F(x-h)]  (2h)  =  0 

h  -  0 

Proof 

It  suffices  given  arbitrary  e  >  0  to  exhibit  a  set  A  with 
ji(F(A))  >  1  -  c  whose  elements  satisfy  a  Holder  condition  of  the 
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k  above  type.  Choose  a  positive  6  <  OC  and  consider  the  set  of  points, 

B,  for  which 

*  +  q)  -  p(x  -  q)  >  (Ci  +  q)H’6 

for  infinitely  many  i.  Cover  B  with  a  countable  collection  of 

intervals,  C.  ,  each  of  length  <  3X  and  so  chosen  that  for  each 
A 

xeB  the  interval 

[x  -  2  Tli  -  q  ,  X  +  q  +  2  q]  c  Cx 

where  i  is  the  smallest  integer  for  which  both  Q  +  r)^  <  A  and 
F(X  +  q)  -  F (x  -  q)  >  (q  +  T)i )H“ 6  . 

Since  /x(F(B))  =  0  we  can  choose  A  so  small  that  /i(F(C  ))  <  € 

A 

v> 

Set  A  =  E  -  C.  .  Clearly  /j(F(A))  >  1  -  e  .  For  each  xeA  we  can 

A 

choose  an  integer  t  so  large  that  +  r|t  <  A  and  for  all  integers 
m  £  t 

F(x  +  Cm>  -  F(x  -  V  <  (Cm  +  ^>“'6  ' 

This  implies  that  B  fl  [x  -  Q  -  2  t)  ,  x  +  2  q  +  q]  =  0  and 
further  that  there  exists  an  integer  g  5:  t  such  that  for  all  s  £  g 

H_  c 

"  F<x  +  2  Cs  +  T1S)  -  F(x  -  q  -  2  ng)  ^  3  (q  +  q) 

9  The  desired  Holder  condition  for  such  points  x  follows  easily  since 

6  <  a  • 
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Remark  4) 


Let  {o^}  be  a  sequence  of  positive  numbers  which  decrease 
monotonically  to  zero.  Let  be  the  set  of  xcE  for 
which  F(x+h)  -  F(x-h)  <  (2h)^’aj  for  all  sufficiently 
small  positive  h.  By  Lemma  5,  /i( F(A^))  =  1.  Clearly 
the  sequence  of  sets  A^  decrease  monotonically  to  a 
set  A  with  /i(F(A))  =  1  and  if  xeA  then  F(x+h)  -  F(x-h) 

<  (2h)  ^  for  any  y  >  0  and  all  sufficiently  small  h  >  0. 


We  now  recall  some  definitions.  The  y  dimensional  Hausdorff 

measure  of  a  set  AC  [0,1]  is  defined  by  r  (y;  A)  =  liminf  £  fi(I . 

5-0  j  J 

where  the  liminf  is  taken  over  all  countable  sets  of  intervals  f I . } 

J 

such  that  (J  1^  c  A  and  p(I  )  <  6  •  The  Hausdorff  dimension  of  the 
set  A,  /3(A),  is  the  number  with  the  property  that  for  any  e  >  0 

T  [/3(A)  -  e,  A]  =  <*>  T  [/3(A)  +  e,  A]  =  0  . 


We  now  prove 
Theorem  2 

If  E*C  E  and  F(E*))  >  0  then  /3(E*)  -  H. 

Proof 

Given  any  fixed  e  >  0  there  exists  for  each  xeE  a  smallest 
i  ^  n  such  that 

H+e 

F(x  +  q)  -  F(x  -  r)t)  >  [q  +  T1-] 

We  can  thus  construct  a  countable  set  of  mutually  disjoint  intervals 
i  -  1,2,*  ••  each  of  length  £  D  n  and  so  chosen  that 

U  I.(n)  3  E*  and  /i(F(I  (n)))  £  /i(I  )H+6  • 

i 
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Thus 

1  £  fi( F(  U  I.(n)))  *  E  (fi(I.(n)))  €  . 

i  i  L 

Hence 

r  (H+e,  E*)  s:  1  and  /3(E*)  £  H  . 

To  prove  the  inequality  in  the  other  direction  fix  e  >  0  and 
choose  Jig  >  0  and  a  set  A(h^)  C  E  so  that  ji(F(A(hQ)))  S  6  >  0 

and  for  each  xeAChg)  and  every  h  £  h^  F(x+h)  -  F(x-h)  £  (2h)^  6  . 

Now  cover  A(h^)  with  a  set  of  intervals  1^,  each  chosen  with  length 
<  h^.  We  can  assume  that  each  interval  1^  contains  at  least  one  point, 
x^sAOIq)  .  Let  be  the  smallest  interval  symmetric  about  x^  which 
contains  1^  .  Then  £  2ji(I^)  and  correspondingly 

E  (2M(I.))H'e  ^  S  <VL(J ^  E  fl(F( J.))  £  /j(f(A(h0)»  :>  6  . 

Since  the  chosen  covering  of  A(hp)  was  arbitrary,  aside  from  the 
fact  that  the  length  of  each  interval  was  constrained  to  be  smaller 
than  hQ  ,  T  (H-e,  A(hQ))  £  6  .  Hence  £  (A ChQ))  ^  H. 

But  E*  3  A(hQ)  thuS  &  (E*)  ^  H* 

Remark  5)  The  proof  as  presented  is  valid  for  0  £  H  £  1 .  The  effort 
expended  is,  however,  only  really  needed  for  distribution 
functions  of  the  second  type.  For  if  F  is  type  (1)  the 
set  E  consists  of  the  k  jump  points  and  if  F  is  type  (3) 

E  =  [0,1].  The  Hausdorff  dimension  of  a  finite  or  even 
Countable  set  of  points  is  0;  whereas  the  Hausdorff  dimension 
of  a  bounded  set  of  positive  Lebesque  measure  is  1. 
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